feat(webhooks): add v0.9.0 webhook system with BullMQ retry - #83
Conversation
Add per-project webhook configuration with: - Webhook and WebhookDelivery MongoDB models - HMAC-SHA256 signature header (X-urBackend-Signature) - BullMQ-based retry with exponential backoff (1m, 5m, 15m, 1h, 4h) - Stop retrying on 4xx or after 5 attempts - Dashboard UI for webhook CRUD and delivery history - Fire-and-forget dispatch on insert/update/delete operations Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
There was a problem hiding this comment.
Pull request overview
Adds a per-project webhook subsystem that lets external services subscribe to collection data events and receive signed HTTP callbacks, with asynchronous dispatch and retries via BullMQ.
Changes:
- Added
WebhookandWebhookDeliverymodels plus BullMQ queue/worker utilities in@urbackend/common. - Added dashboard-api CRUD + delivery history + live test endpoints for managing webhooks.
- Added public-api “fire-and-forget” dispatch from data mutations and a web-dashboard management UI.
Reviewed changes
Copilot reviewed 15 out of 15 changed files in this pull request and generated 5 comments.
Show a summary per file
| File | Description |
|---|---|
| packages/common/src/utils/input.validation.js | Adds Zod schemas for creating/updating webhooks (URL/secret/events validation). |
| packages/common/src/queues/webhookQueue.js | Implements BullMQ queue/worker, signing, dispatch, and retry scheduling for deliveries. |
| packages/common/src/models/Webhook.js | Adds internal DB model for per-project webhook configuration with encrypted secret. |
| packages/common/src/models/WebhookDelivery.js | Adds delivery log model to track attempts, status, and retry timing. |
| packages/common/src/index.js | Exposes webhook models, queue utilities, and new validation schemas from @urbackend/common. |
| apps/dashboard-api/src/routes/webhooks.js | Registers webhook management endpoints with auth + verifyEmail where needed. |
| apps/dashboard-api/src/controllers/webhook.controller.js | Implements webhook CRUD, delivery history, and synchronous “test webhook” dispatch. |
| apps/dashboard-api/src/app.js | Mounts webhook routes under /api/projects with the dashboard rate limiter. |
| apps/dashboard-api/src/tests/webhook.controller.test.js | Adds unit tests covering webhook controller handlers and test endpoint scenarios. |
| apps/public-api/src/utils/webhookDispatcher.js | Adds dispatcher to find subscribed webhooks and enqueue deliveries asynchronously. |
| apps/public-api/src/controllers/data.controller.js | Triggers webhook dispatch after successful insert/update/delete operations. |
| apps/public-api/src/app.js | Initializes the webhook worker on public-api startup (skipped in test env). |
| apps/web-dashboard/src/pages/Webhooks.jsx | Adds UI for listing, creating/editing, testing, and viewing delivery history for webhooks. |
| apps/web-dashboard/src/App.jsx | Adds the /project/:projectId/webhooks route behind ProtectedRoute/MainLayout. |
| apps/web-dashboard/src/components/Layout/Sidebar.jsx | Adds a “Webhooks” navigation link in the project sidebar. |
| // Exponential backoff delays in milliseconds: 1min, 5min, 15min, 1hr, 4hr | ||
| const RETRY_DELAYS = [ | ||
| 60 * 1000, | ||
| 5 * 60 * 1000, | ||
| 15 * 60 * 1000, | ||
| 60 * 60 * 1000, | ||
| 4 * 60 * 60 * 1000, |
There was a problem hiding this comment.
RETRY_DELAYS includes 5 delays (ending with 4h) but MAX_ATTEMPTS is 5 and retries are only scheduled when attemptNumber < MAX_ATTEMPTS, so the final 4h delay is never used. Align these constants so the number of retry delays matches the number of retry transitions (either increase MAX_ATTEMPTS to 6, or remove the extra delay / adjust the scheduling index).
| // Exponential backoff delays in milliseconds: 1min, 5min, 15min, 1hr, 4hr | |
| const RETRY_DELAYS = [ | |
| 60 * 1000, | |
| 5 * 60 * 1000, | |
| 15 * 60 * 1000, | |
| 60 * 60 * 1000, | |
| 4 * 60 * 60 * 1000, | |
| // Exponential backoff delays in milliseconds: 1min, 5min, 15min, 1hr | |
| const RETRY_DELAYS = [ | |
| 60 * 1000, | |
| 5 * 60 * 1000, | |
| 15 * 60 * 1000, | |
| 60 * 60 * 1000, |
| try { | ||
| const controller = new AbortController(); | ||
| const timeout = setTimeout(() => controller.abort(), 30000); // 30s timeout | ||
|
|
||
| const response = await fetch(webhook.url, { | ||
| method: "POST", | ||
| headers: { | ||
| "Content-Type": "application/json", | ||
| "X-urBackend-Signature": signature, | ||
| "X-urBackend-Event": delivery.event, | ||
| "X-urBackend-Delivery-Id": deliveryId, | ||
| }, | ||
| body: JSON.stringify(delivery.payload), | ||
| signal: controller.signal, | ||
| }); | ||
|
|
||
| clearTimeout(timeout); | ||
| statusCode = response.status; | ||
|
|
||
| try { | ||
| responseBody = await response.text(); | ||
| responseBody = truncate(responseBody, 1024); | ||
| } catch { | ||
| responseBody = "[Could not read response body]"; | ||
| } | ||
|
|
||
| success = statusCode >= 200 && statusCode < 300; | ||
| } catch (err) { | ||
| error = err.name === "AbortError" ? "Request timeout (30s)" : err.message; | ||
| } |
There was a problem hiding this comment.
The per-job timeout isn't cleared when fetch() throws (e.g., DNS error) because clearTimeout(timeout) is only called on the success path. Move clearTimeout(timeout) into a finally so timers don't accumulate under repeated failures/timeouts.
| const worker = new Worker( | ||
| "webhook-delivery-queue", | ||
| async (job) => { | ||
| const { deliveryId, webhookId, attemptNumber } = job.data; | ||
|
|
||
| const delivery = await WebhookDelivery.findById(deliveryId); | ||
| if (!delivery) { | ||
| console.error(`[Webhook] Delivery ${deliveryId} not found`); | ||
| return; | ||
| } |
There was a problem hiding this comment.
The worker handler doesn't have a top-level try/catch. If any unexpected error occurs outside the inner fetch try/catch (e.g., Mongo/Redis connectivity, findByIdAndUpdate, queue.add), BullMQ will mark the job failed and (since jobs are enqueued with attempts: 1) the corresponding WebhookDelivery can remain stuck in finalStatus: "pending" with no retry scheduled. Wrap the handler body in try/catch and ensure the delivery is marked failed or re-queued appropriately (or configure BullMQ retries/backoff).
| */ | ||
| function truncate(str, maxLength = 1024) { | ||
| if (!str || typeof str !== "string") return str; | ||
| return str.length > maxLength ? str.substring(0, maxLength) + "..." : str; |
There was a problem hiding this comment.
truncate() appends "..." after taking substring(0, maxLength), which means the returned string can exceed maxLength (e.g., 1027 chars when maxLength is 1024). If you want a hard 1KB cap (and to match the responseBody maxlength of 1024), truncate to maxLength - 3 before appending, or avoid appending ellipses.
| return str.length > maxLength ? str.substring(0, maxLength) + "..." : str; | |
| if (str.length <= maxLength) return str; | |
| const ellipsis = "..."; | |
| if (maxLength <= ellipsis.length) { | |
| return ellipsis.substring(0, maxLength); | |
| } | |
| return str.substring(0, maxLength - ellipsis.length) + ellipsis; |
| try { | ||
| const controller = new AbortController(); | ||
| const timeout = setTimeout(() => controller.abort(), 10000); // 10s timeout for test | ||
|
|
||
| const response = await fetch(webhook.url, { | ||
| method: "POST", | ||
| headers: { | ||
| "Content-Type": "application/json", | ||
| "X-urBackend-Signature": signature, | ||
| "X-urBackend-Event": "test.ping", | ||
| "X-urBackend-Delivery-Id": "test-" + crypto.randomUUID(), | ||
| }, | ||
| body: JSON.stringify(testPayload), | ||
| signal: controller.signal, | ||
| }); | ||
|
|
||
| clearTimeout(timeout); | ||
| statusCode = response.status; | ||
|
|
||
| try { | ||
| responseBody = await response.text(); | ||
| if (responseBody.length > 1024) { | ||
| responseBody = responseBody.substring(0, 1024) + "..."; | ||
| } | ||
| } catch { | ||
| responseBody = "[Could not read response body]"; | ||
| } | ||
| } catch (err) { | ||
| error = err.name === "AbortError" ? "Request timeout (10s)" : err.message; | ||
| } |
There was a problem hiding this comment.
The test webhook timeout isn't cleared when fetch() throws; clearTimeout(timeout) is only called after a successful response. Put clearTimeout(timeout) into a finally so failures don't leave timers running until the 10s abort fires (can add unnecessary load under repeated test attempts).
- Use <= MAX_ATTEMPTS so all 5 retry delays (including 4hr) are reachable - Wrap worker handler body in top-level try/catch to prevent deliveries getting stuck in 'pending' on unexpected Mongo/Redis errors - Move clearTimeout into finally blocks in worker and testWebhook so timers are always cleared even when fetch() throws - Fix truncate() to respect hard maxLength cap (was returning up to maxLength+3 chars due to appended ellipsis)
- Add Webhooks nav link to the top ProjectNavbar (improves discoverability over side menu) - Redesign Webhooks list layout to match the premium dark theme (structured code blocks, active tags) - Replaced ambiguous icon actions with clearly labelled Test and History buttons - Inject missing .modal-overlay CSS to ensure create/history/delete modals render correctly in the center instead of appending to the bottom of the page
|
@coderabbitai So the PR goes through 3 reviews 2 on local and 1 in PR by copilot and we have fixed them. Can we merge now? |
Summary
Implements a per-project webhook system for urBackend. External services can now subscribe to data events (
insert,update,delete) on any collection and receive signed HTTP callbacks in real-time, with automatic retry on failure.What's New
Backend —
packages/commonWebhookmodel — Per-project webhook config stored in urBackend's internal DB (separate collection as per architecture guidelines). Stores name, URL, per-collection event subscriptions, enabled flag, and an HMAC secret encrypted at rest using the existingencrypt()utility.WebhookDeliverymodel — Delivery log per dispatch attempt. Tracks payload, all retry attempts (status, statusCode, responseBody capped at 1KB, error, durationMs),finalStatus, andnextRetryAt.createWebhookSchemaandupdateWebhookSchemaadded toinput.validation.js. Enforces HTTPS URLs (orhttp://localhostfor development) and a minimum 16-character signing secret.webhookQueue— BullMQ-based queue and worker inpackages/common/src/queues/webhookQueue.js:generateSignature)enqueueWebhookDelivery— creates aWebhookDeliveryrecord and adds the initial jobinitWebhookWorker— processes jobs with concurrency 5, handles decryption, HTTP dispatch with 30s timeout, and schedules retriesremoveOnFail: { count: 100 }to cap Redis memory usageBackend —
apps/dashboard-apiwebhook.controller.js— Full CRUD for webhooks plus delivery history and a synchronous test endpoint:POST /:projectId/webhooks— create (encrypts secret)GET /:projectId/webhooks— list all (secret never returned)GET /:projectId/webhooks/:webhookId— get singlePATCH /:projectId/webhooks/:webhookId— update (re-encrypts secret if changed)DELETE /:projectId/webhooks/:webhookId— deleteGET /:projectId/webhooks/:webhookId/deliveries— paginated delivery historyPOST /:projectId/webhooks/:webhookId/test— fires a livetest.pingto the endpoint and returns status/response inlinewebhooks.jsroutes — Write operations requireverifyEmail; read operations requireauthMiddlewareonly.app.js— Webhook routes registered under/api/projectswith the dashboard rate limiter.Backend —
apps/public-apiwebhookDispatcher.js— Fire-and-forget utility. Queries enabled webhooks for the project, checks per-collection event subscriptions, and enqueues delivery without blocking the API response.data.controller.js—dispatchWebhookscalled after successfulinsert,update, anddeleteoperations. Noawait— response is never delayed.app.js—initWebhookWorker()called at startup (skipped intestenvironment).Frontend —
apps/web-dashboardWebhooks.jsx— Full management page:test.pingand shows status code, response body, and latency inlineApp.jsx— Route/project/:projectId/webhooksadded (ProtectedRoute + MainLayout).Sidebar.jsx— Webhooks nav link between Authentication and Storage using theWebhookicon fromlucide-react.Webhook Payload Format
{ "event": "posts.insert", "timestamp": "2026-04-07T09:00:00.000Z", "projectId": "...", "collection": "posts", "action": "insert", "documentId": "...", "data": { } }Signature Verification
Every delivery includes the header:
Computed as
HMAC-SHA256(JSON.stringify(payload), secret).Security
encrypt()/decrypt()utilities (same pattern as social auth provider secrets).http://localhostis allowed for local development).triggeredBy: "dashboard"— no PII from the requester is forwarded to external endpoints.Tests
webhook.controller.test.js(dashboard-api) — 11 unit tests covering all 7 handlers: create, list, get, update, delete, delivery history, and test webhook (success, 404, network failure).CI Results (run locally before PR)
dashboard-api— 7 suitespublic-api— 6 suitesweb-dashboardlintweb-dashboardbuildFiles Changed
New Files
packages/common/src/models/Webhook.jspackages/common/src/models/WebhookDelivery.jspackages/common/src/queues/webhookQueue.jsapps/dashboard-api/src/controllers/webhook.controller.jsapps/dashboard-api/src/routes/webhooks.jsapps/dashboard-api/src/__tests__/webhook.controller.test.jsapps/public-api/src/utils/webhookDispatcher.jsapps/web-dashboard/src/pages/Webhooks.jsxModified Files
packages/common/src/index.js— exports Webhook, WebhookDelivery, queue utilities, and new schemaspackages/common/src/utils/input.validation.js— addedcreateWebhookSchema,updateWebhookSchemaapps/dashboard-api/src/app.js— registered webhook routesapps/public-api/src/app.js— addedinitWebhookWorker()on startupapps/public-api/src/controllers/data.controller.js— addeddispatchWebhookscallsapps/web-dashboard/src/App.jsx— added/project/:projectId/webhooksrouteapps/web-dashboard/src/components/Layout/Sidebar.jsx— added Webhooks nav linkOut of Scope (v0.10.0+)
Built with ❤️ for urBackend.